The time-based data from a real-time feed is typically being transferred to a database such as a relational data base management system, RDBMS. One example of such a real-time stream of data would be quotes on stocks being traded. Within the RDBMS, data is organized such that it can be easily found. For example, if a user were to request a quote for IBM stock, the price would be quickly found. However, the real-time stream of data is initially stored in a memory buffer then transferred to disk memory. With current data streaming technologies, there exists a relatively long lag time between the time the data arrives in the buffer to the time it is available for use by the RDBMS.
Many applications need the fast response that is typical of in-memory databases, but also need the reliability and recoverability of traditional disk based relational database management systems. What these applications require is a method for copying data from the in-memory database to the on-disk RDBMS, such that the data can be easily recovered in the event of a machine crash.
Minimizing the lag time between the data written to the in-memory database and subsequently written to the RDBMS is critical. This lag should be as short as possible to minimize the loss of data in the event of a machine crash. In addition, if the data copied to the RDBMS is time stamped then the RDBMS effectively represents the state of the in-memory database over time. This allows for the possibility of recreating the in-memory database from the RDBMS for any point in time.
Conventional technologies that store streaming data in a buffer memory and then transfer this data to a database have relatively long lag times between the arrival of the data and the transfer to the database. What is therefore needed is a system and an associated method for reducing this lag time. The need for such system and method has heretofore remained unsatisfied.